by Ray Cai
April 15, 2017
Terminology:
Neuron
perceptron
Neural Network
The distance between actual result and prediction.
Any distance would work, the ordinary euclidian distance is fine but for classification problems one distance, called the "cross-entropy" is more efficient.
Y=softmax(X.W+b)
cross−entropy=−∑Y​i​′​​.logY​i​​
"Training" the neural network actually means using training images and labels to adjust weights and biases so as to minimise the cross-entropy loss function.
import tensorflow as tfX = tf.placeholder(tf.float32, [None, 28, 28, 1])W = tf.Variable(tf.zeros([784, 10]))b = tf.Variable(tf.zeros([10]))init = tf.initialize_all_variables()
# modelY = tf.nn.softmax(tf.matmul(tf.reshape(X, [-1, 784]), W) + b)# placeholder for correct labelsY_ = tf.placeholder(tf.float32, [None, 10])# loss functioncross_entropy = -tf.reduce_sum(Y_ * tf.log(Y))# % of correct answers found in batchis_correct = tf.equal(tf.argmax(Y,1), tf.argmax(Y_,1))accuracy = tf.reduce_mean(tf.cast(is_correct, tf.float32))
optimizer = tf.train.GradientDescentOptimizer(0.003)train_step = optimizer.minimize(cross_entropy)
sess = tf.Session()sess.run(init)for i in range(1000):# load batch of images and correct answersbatch_X, batch_Y = mnist.train.next_batch(100)train_data={X: batch_X, Y_: batch_Y}# trainsess.run(train_step, feed_dict=train_data)# success ?a,c = sess.run([accuracy, cross_entropy], feed_dict=train_data)# success on test data ?test_data={X: mnist.test.images, Y_: mnist.test.labels}a,c = sess.run([accuracy, cross_entropy], feed=test_data)